The management of security information and security events has progressed into two approaches. One approach is the security information and event management (SIEM) system that collects security related information into a relational database and allows the analyst to analyze events by querying a database. A second technique is by indexing log messages, allowing predefined signatures to trigger on incoming events, and having an analyst then search the indexed logs for details. These approaches to managing security information and events are limited to only alerts and system log messages, which is a small subset of the overall activity of a network.
Security information and event management (SIEM) leverages high-speed relational databases. In a relational database, when an event occurs (i.e., something that is known to be bad or suspicious), the database is queried to validate that an incident has occurred. In a SIEM process (such as depicted in FIG. 2): (a) log events may be added to a event log or table; (b) reputation and validation engines review incoming information to generate a new record in a related table; and (c) queries are performed over the joined tables and results are given to a user to view or correlate. A SIEM may use a transaction table for handling flow data. A transaction table is a table that inserts events directly into the table without modification. The information may be cross-indexed and then joined when the user searches and fetches data. In the SIEM process, data is not preprocessed before storage, just merely indexed. These processes do not merge different data sources into a single table.
Security information management has generally involved leveraging a database to store event data and then joining the known reputational and analysis data. The process of using relational joins becomes apparent as the correlation rules of these systems are defined as structured query language (SQL) operations leveraging joins.
In a flow process (as depicted in FIG. 3): (a) log events may be added to a event log or table unique from a flow log or table; (b) flow events may be added to a flow log or table unique from the event log or table; (c) reputation and validation engines review incoming information to generate new records in a related table; and (d) queries are performed over the joined tables and results are given to the user to view or correlate.
Flow analytics also may be employed to see events that were not detected by other log producers. Such an example is SRI MetaFlows (http://www.metaflows.com/) and Lancope (http://www.lancope.com/products/stealthwatch-system/). Commercially, deep packet inspection engines, like RSA/NetWitness and Solara, are flow analytic systems, including RSA Security Analytics and Solara DeepSee respectively. These systems generate their flow data from the packet captures and then present related knowledge to the user. These approaches are very similar to the relational database approach as they are joining tables of data.
Flow data techniques were the backbone of the Government's Einstein Project. It was based on the flow analysis research done by Carnegie Mellon's Computer Emergency Response Team (CERT). This is the System for Internet-Level Knowledge (http://tools.netsa.cert.org/silk/). The merger of SILk and Snort Alerts (intrusion detection alerts) was an earlier implementation of this approach.